A Topic Coverage Approach to Evaluation of Topic Models

Topic models are widely used unsupervised capable of learning topics - weighted lists words and documents from large collections text documents. When topic for discovery in collections, a question that arises naturally is how well the model-induced correspond to interest analyst. In this paper we revisit extend so far neglected approach model evaluation based on measuring coverage computationally matching with set reference expected uncover. The suited analyzing models' performance large-scale analysis both measures quality. We propose new evaluate, series experiments, different types two distinct domains which exists. experiments include quality, categories, relationship between other methods evaluation. contributes supervised measure coverage, first coverage. achieves accuracy close human agreement. correlates highly one (Spearman's $\rho \geq 0.95$). Other contributions insights into evaluation, datasets code facilitating future research